Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stratum v2 Template Provider via IPC interface (multiprocess) #48

Open
wants to merge 35 commits into
base: 2024/07/sv2-tp-common
Choose a base branch
from

Conversation

Sjors
Copy link
Owner

@Sjors Sjors commented Jul 17, 2024

This is an alternative approach to #68. Rather than integrating the Template Provider directly into Bitcoin Core, we make it an entirely separate application that communicates via IPC, see design/multiprocess.md.

Usage

Compile as usual, multiprocess is enabled by default.

Alternatively get a recent binary, see (unofficial, by me) releases ending in "ICP".

Start node:

build/src/bitcoin-node -chain=testnet4 -ipcbind=unix

Start the Template Provider:

build/src/bitcoin-mine -chain=testnet4 -debug=sv2 -loglevel=sv2:trace

See general Stratum v2 usage instructions in bitcoin#29432.

Rationale, PRs this builds on

There are multiple ways to create such an application. It could be built from scratch in Rust, using bits and pieces of the Stratum Reference Implementation SRI. It could be a c++ application built from scratch, perhaps using some libraries from Bitcoin Core.

But this PR goes about in a slightly different way. Similar to Elements or Knots it's a set of patches on top of the full Bitcoin Core codebase. This allows me to leverage the build system, subclass Transport and use various other useful bits.

This lets me reuse almost all code from the integrated Template Provider bitcoin#29432. Both build direct on:

This PR relies on the interface and build changes listed in bitcoin#31098. These interface changes are also used by bitcoin#29432, but there they run in the same process.

I modified the guix script to only produce bitcoin-node (for now) and bitcoin-mine binaries.

Goal

The goal is to drop bitcoin-node from the release here. Ideally users would install Bitcoin Core in the manner they're familiar with. They would then install bitcoin-mine separately and it should just work(tm), at most having to add a line to bitcoin.conf to turn IPC on.

This won't work until we ship a release with multiprocess and the mining interface enabled, ideally in bitcoind and bitcoin-qt rather than in seperate bitcoin-node and bitcoin-gui binaries. See bitcoin#30437 for progress towards that.

Rebase hint

Rebase hint:

git reset --hard 2024/07/sv2-tp-common
git merge 2025/02/ipc-yea
git cherry-pick sjors/sv2-ipc~14^..sjors/sv2-ipc

@Sjors
Copy link
Owner Author

Sjors commented Jul 17, 2024

@ryanofsky I slightly mangled e373642 because of bitcoin@4e1a434#r1681193702. I'll clean that up later.

@Sjors
Copy link
Owner Author

Sjors commented Jul 17, 2024

Incorporated the changes from bitcoin#30437 (comment) and cleaned up commit history.

@Sjors
Copy link
Owner Author

Sjors commented Jul 18, 2024

@ryanofsky do you have a commit somewhere that adds libmultiprocess to the guix build?

@Sjors Sjors force-pushed the sv2-ipc branch 2 times, most recently from c75792b to 91720b7 Compare July 18, 2024 09:26
@Sjors
Copy link
Owner Author

Sjors commented Jul 18, 2024

Getting a cryptic error: '__NVCC___WORKAROUND_GUARD' is not defined error for the multiprocess job: https://cirrus-ci.com/task/4940413969104896?logs=ci#L2225

Somehow related to the Template Provider depending on boost/multi_index via its use of the mempool.

I can't make sense of the personality(old_personality | ADDR_NO_RANDOMIZE) error thrown by MSAN and TSAN, but it's possible that's related to my self-hosted CI setup and/or the modification in 3319668.

@Sjors Sjors force-pushed the 2024/07/sv2-tp-common branch 2 times, most recently from db58076 to 1ff4660 Compare July 18, 2024 12:24
@Sjors Sjors force-pushed the sv2-ipc branch 3 times, most recently from 4354e96 to 3aa6f0a Compare July 18, 2024 13:11
@Sjors Sjors force-pushed the 2024/07/sv2-tp-common branch from 1ff4660 to ee010f5 Compare July 18, 2024 13:16
@ryanofsky
Copy link

re: #48 (comment)

@ryanofsky do you have a commit somewhere that adds libmultiprocess to the guix build?

I don't, since I haven't really experimented with the guix build. Maybe simplest approach to start with would enable the multiprocess build option and build and package all the multiprocess executables including bitcoin-tp and bitcoin-node. I know in the longer run you want to have separate releases as described in bitcoin#30437 (comment).

re: #48 (comment)

Getting a cryptic error: '__NVCC___WORKAROUND_GUARD' is not defined error for the multiprocess job: https://cirrus-ci.com/task/4940413969104896?logs=ci#L2225

Somehow related to the Template Provider depending on boost/multi_index via its use of the mempool.

This is a -Wundef error caused by bitcoin#29876 that happens when a boost header is included from a source file that is compiled with -I /path/to/boost/headers rather than -isystem /path/to/boost/headers. The fix for this is probably:

-bitcoin_mine_CPPFLAGS = $(bitcoin_bin_cppflags)
+bitcoin_mine_CPPFLAGS = $(bitcoin_bin_cppflags) $(BOOST_CPPFLAGS)

to add the right -isystem option.

I can't make sense of the personality(old_personality | ADDR_NO_RANDOMIZE) error thrown by MSAN and TSAN, but it's possible that's related to my self-hosted CI setup and/or the modification in 3319668.

Not sure about this error. Link seems to be https://cirrus-ci.com/task/4759864851824640?logs=ci#L2486

@Sjors Sjors force-pushed the 2024/07/sv2-tp-common branch 2 times, most recently from 19cc9f1 to dabc32c Compare July 19, 2024 08:59
@Sjors Sjors force-pushed the sv2-ipc branch 3 times, most recently from 6c24b45 to a8bb152 Compare July 19, 2024 13:55
@Sjors
Copy link
Owner Author

Sjors commented Jul 19, 2024

I'm getting the MSAN / TSAN error inconsistently on my self-hosted CI. Does not seem related to this PR. Trying to reproduce in isolation in #51.

ryanofsky and others added 29 commits February 10, 2025 10:03
When ENABLE_IPC option is on, build with libmultiprocess subtree and
`add_subdirectory(src/ipc/libmultiprocess)` instead of external package
and `find_package(Libmultiprocess)` by default.

Behavior can be toggled with `WITH_LIBMULTIPROCESS` option. Using a subtree
should be more convenient for most bitcoin developers, but using an external
package is more convenient for developing in the libmultiprocess repository.

The `WITH_LIBMULTIPROCESS` option is also used to avoid needing to changing the
depends build here. But in later commits, the depends build is switched to use
the add_subdirectory build as well.

Co-authored-by: Cory Fields <[email protected]>
This change is technically not needed to add libmultiprocess as a subtree, but
it avoids a CI failure in followup PR bitcoin#30975 which enables multiprocess build
option in more CI jobs. In that PR, several jobs fail due to the mptest
executable not being built by default, as reported
bitcoin#30975 (comment)
This change is technically not needed to add libmultiprocess as a subtree, but
it avoids a CI failure in followup PR bitcoin#30975 which enables multiprocess build
option in more CI jobs. In that PR, the "macOS 14 native no depends job" fails
due to warnings in boost headers treated as errors, reported in
bitcoin#30975 (comment) and
chaincodelabs/libmultiprocess#138
This change is technically not needed to add libmultiprocess as a subtree, but
it avoids a CI failure in followup PR bitcoin#30975 which enables multiprocess build
option in more CI jobs. In that PR, clang-tidy job fails due to missing
generated example files as reported
bitcoin#30975 (comment)

Different fixes were suggested
bitcoin#30975 (comment) and
bitcoin#30975 (comment)

Co-authored-by: Hennadii Stepanov <[email protected]>
Without this change linter produces errors about:

- Use of std::filesystem the libmultiprocess example program.
- Use of locale-dependent functions in example program, in the build time code
  generator, and in the runtime library for debug logging.
- Include guards not beginning with BITCOIN_
Move parts of the int_get_build_id into a new int_get_build_properties
function. There is no change in behavior. This just organizes assignments
better so some build properties can be used to help compute build ids in the
next commit.
With newly introduced libmultiprocess subtree, there's no need for depends
system to download and track changes to the upstream repository.

Note that adding the libmultiprocess subtree does not allow dropping
libmultiprocess packages from the depends build, because libmultiprocess
includes a code generation tool called mpgen, and in cross-compiled builds,
bitcoin core's cmake build system doesn't have access to a native toolchain and
can't build mpgen itself, so the depends system (or the native environment if
not using depends) needs to supply it.
Except:
1. i686, DEBUG (changed to "no multiprocess")
2. Windows due to lack of support
Install capnp where needed.

The bitcoin-node binary is built on all platforms which have
multiprocess enabled, but for functional tests it's only used in
the macOS native (no depends) and CentOS native (depends) jobs.
Except for Windows and OpenBSD.
Allows InitError and InitWarning to be used by other executables beside
bitcoind and bitcoin-node.
See src/bitcoin-mine.cpp for usage information.

Co-authored-by: Sjors Provoost <[email protected]>
For now also build bitcoin-node.
The Template Provider binary:
* won't have a wallet
* probably won't have a gui
* doesn't use the functional test framework

Disable some CI stuff to speed it up.

This is not a complete list, because:
1. That would be too hard to rebase
2. Some things might be added later, e.g. tracepoints
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants